SPECIFICATION 



TITLE OF THE INVENTION 
APPARATUS FOR AND METHOD OF CALCULATING LENS 
DISTORTION FACTOR, AND COMPUTER READABLE STORAGE 
MEDIUM HAVING LENS DISTORTION FACTOR 
CALCULATION PROGRAM RECORDED THEREON 



BACKGROUND OF THE INVENTION 



Field of the Invention 

The present invention relates to an apparatus 
for and a method of calculating a lens distortion 
factor, a computer readable recording medium having 
a lens distortion factor calculation program 
recorded thereon, an apparatus for and a method of 
combining images, and a recording medium having an 
image constructing program recorded thereon. 
Description of the Prior Art 

[1] Description of Conventional Method of 
Calculating Optical Flow 

A technique for calculating an optical flow 
from two images, and registering the two images on 
the basis of the obtained optical flow has been 
known. Description is made of a conventional method 



of calculating an optical flow. 
(1) Lucas-Kanade Method 

A large number of methods for calculating the 
apparent optical flow of a moving object in a moving 
image have been conventionally proposed. The 
Lucas-Kanade method which is a local gradient method 
out of the methods is one of the best methods. The 
reason for this is that the speed of processing is 
high, implementing is easy, and the result has 
confidence . 

As for the details of the Lucas-Kanade method, 
see an article entitled by B . Lucas and T. Kanade, 
"An Iterative Image Registration Technigue with an 
Application to Stereo Vision", In Seventh 
International Joint Conference on Artificial 
Intelligence (IJCAI-81), pp. 674-979, 1981. 

The outline of the Lucas-Kanade method will be 
described below. 

When a gray scale pattern I (x, y, t) of image 
coordinates p = (x, y) at time t is moved to 
coordinates (x + Sx, y + 8 y) with its gradation 
distribution kept constant after a very small time 
period ( 5 t) , the following optical flow constraint 
equation (1) holds: 
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In order to calculate an optical flow {v = ( 5 
x/5t,Sy/St) = ( u , v ) } in a two-dimensional image, 
another constraint equation is required because the 
number of unknown parameters is two. Lucas and 
Kanade have assumed that the optical flow is constant 
in a local region of an object. 

Suppose the optical flow is constant within a 
local region co on an image, for example. In this 
W case, a squared error E of a gray scale pattern which 

is to be minimized can be defined by the following 
equation (2) when the following substitutions are 
made: 

I o (p) = I ( x , y f t ) , 
ii (p+v) = i(x+u, y+v r t+St) 
E = Y*[I,(p + v)-I 0 (p)l 3 - (2) 

When v is very small, the terms of second and 
higher degrees in Tailor series expansion can be 
ignored. Therefore, a relationship expressed by the 
following equation (3) holds: 
//p + vj = //pj+gfp>v ( 3 ) 

where g(p) is a linear differential of Ii (p) . 
The error E is minimized when the derivative of 
E with respect to v is zero. Therefore, a 
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relationship expressed by the following equation (4) 
holds ; 

8v 

= £2gfp;[//p;+gfpjv-/ 0 f P ;] 

Therefore, the optical flow v is found by the 
following equation (5) : 

_. E.*(p<W-VpJ] 

Furthermore, the optical flow can be found with 
high precision by Newton-Raphson iteration, as 
expressed by the following equation (6) : 

v =v + LUw:] 

^=^(P + Vj - (6) 

g„ =8(P + v k ), 

I 0 =h(p) 

(2) Hierarchical Estimation Method 
The largest problem of the gradient methods, 
including the Lucas-Ranade method, is that the 
methods cannot be applied to a large motion because 
a good initial value is required. Therefore, a 
method of producing images respectively having 
resolutions which differ at several levels like a 
pyramid hierarchical structure to solve the problem 
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has been conventionally proposed. 

Images having resolutions which differ at 
several levels are first previously produced from 
each of two consecutive images. An approximate 
optical flow is then calculated between the images 
having the lowest resolution. A more precise optical 
flow is c a 1 c u 1 a t e d b e t w e e n images h a v i n g r e s o 1 u t i o n s 
which are higher by one level with reference to the 
results. The processing is successively repeated 
until the optical flow is calculated between the 
images having the highest resolution. 

Fig. 4 , Fig. 3, Fig. 2 , and Fig. 1 respectively 
illustrate an original image, an image having a lower 
resolution than that of the original image shown in 
Fig. 4, an image having a lower resolution than that 
of the image having a low resolution shown in Fig. 
3, and an image having a lower resolution than that 
of the image having a low resolution shown in Fig. 
2. In Figs. 1 to 4, S denotes one patch. 

An optical flow is gradually found from the 
image shown in Fig. 1 (an image in a hierarchy 1) , 
the image shown in Fig. 2 (an image in a hierarchy 

2) , the image shown in Fig. 3 (an image in a hierarchy 

3) , and the image shown in Fig. 4 (an image in a 
hierarchy 4) in this order. In Figs. 1 to 4, an arrow 
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indicates an optical flow vector found for each 
patch . 

However, the problem herein is that in a real 
image, there are few regions containing sufficient 
texture, so that a reliable optical flow is not 
obtained . 

[2] Description of Optical Flow Calculating 
Method Developed by Applicant of Present Invention 

An optical flow calculating method developed by 
the applicant of the present invention presupposes 
hierarchical prediction for producing images having 
resolutions which differ at several levels like a 
pyramid hierarchical structure to gradually 
calculate an optical flow. A method of calculating 
an optical flow conforms to a gradient method such 
as the Lucas-Kanade method. That is, it presupposes 
an optical flow estimation method using a 
hierarchically structured gradient method. The 
Lucas-Kanade method is used as the gradient method. 

The optical flow estimation method developed by 
the applicant of the present invention is 
characterized in that an optical flow obtained in 
each of stages of the optical flow estimation method 
using the hierarchically structured Lucas-Kanade 
method is filled by dilation processing. This will 



be described in detail below. 

One of advantages of the Lucas-Kanade method is 
that the result of tracking has confidence. Tomasi 
and Kanade have showed that the trackability of a 
region can be calculated from image derivatives as 
follows (C. Tomasi and T . Kanade, n Shape and Motion 
from Image Streams; a Factorisation method-Part 3 
Detection and Tracking of Point Features", CMU- 
CS-91-132 , Carnegie Mellon University, 1991) . 

From a 2 X 2 matrix of coefficients G in the 
following equation (7) having as elements the 
squares of differentials in the vertical and 
horizontal directions of an image a) in a region, its 
eigenvalues are calculated, thereby making it 
possible to determine the trackability of the 
region : 

G = £g(p^C P / - (7) 

When both the eigenvalues of the matrix G are 
large, the region changes in orthogonal directions, 
and can be uniquely positioned. Consequently, the 
confidence y of the result of tracking can be 
obtained by the following equation (8) from the 
smaller eigenvalue X min and a gray scale residual E 
between the regions after the tracking: 



The inventors and others of the present 
invention have developed a method of interpolating 
a region having low confidence using the result 
having high confidence in the same hierarchy in an 
optical flow. This uses the result in a hierarchy 
at the next coarser resolution level for only an 
initial value for tracking and does not utilize the 
result in a hierarchy at the current resolution level 
which is paid attention to. Alternatively, it is 
assumed that an optical flow in a region which is 
hardly textured has a value close of optical flows 
in its surrounding regions, to fill a flow field by 
morphology processing. 

Fig. 5 shows how flow vector dilation 
processing is performed. 

The left drawing indicates a map of the 
confidence of a flow vector on a gray scale. It is 
assumed that the blacker the map is, the higher the 
confidence is. 

Obtained flow is first subjected to threshold 
processing. A white portion is subjected to 
threshold processing because the result has low 
confidence . 
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The dilation processing of the result in the 
flow field is then performed as follows in imitation 
of hole filling processing using a morphology 
operation in a binary image. A flow vector u (i, 
j) in coordinates i, j of a region can be calculated, 
as expressed by the following equation (9), upon 
being weighted depending on confidence y from flow 
vectors in its four vicinities: 

*(hj) = Y, Z (9) 

P>Q i A 



'fi | 



(p,q) = (0,1), (0-1), (-1,0)(1,0) 

The processing is repeated until all regions 
having low confidence which have been subjected to 
threshold processing are filled. The filling 
processing is performed in each hierarchy. The flow 
vector u(i, j) in the coordinates i, j of the region 
may be calculated upon being weighted depending on 
confidence y from flow vectors in its eight 
vicinities . 

Fig. 6a shows optical flow which has been 
subjected to threshold processing for an image in 
a hierarchy, and Fig. 6b shows optical flow after 
filling. In Fig. 6a, a r r o w s a r e o p t i c a 1 f 1 o w v e c t o r s 
whose confidence is judged to be high by the 
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threshold processing, and X marks indicate portions 
whose confidence is judged to be low. 

[3] Description of Conventional Method of 
Producing Panoramic Image 

The applicant and others of the present 
invention have already developed a method of 
producing a panoramic image by calculating geometric 
parameters between a plurality of images and 
composing the images on the basis of the calculated 
geometric parameters (see J P - A - 1 1 - 3 3 9 0 2 1 ) . 

This time, the applicant of the present 
invention has proposed a lens distortion calibrating 
method in order to obtain a higher precision 
panoramic image. In a conventional lens distortion 
calibrating method, a special calibrating pattern 
has been prepared, to pick up a plurality of images 
with the calibrating pattern, and calculate a lens 
distortion factor on the basis of obtained images 
(a document entitled by Z. Zhang 'Flexible camera 
calibration by viewing a plane from unknown 
orientations Proc. ICCV99 . pp. 666-673, 1999). 

However, it is difficult for a user to calibrate 
lens distortion using a calibrating pattern. 

SUMMARY OF THE INVENTION 
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An object of the present invention is to provide 
a lens distortion factor calculating apparatus, a 
lens distortion factor calculating method, and a 
computer readable recording medium having a lens 
distortion factor calculation program recorded 
thereon, in which lens distortion can be calibrated 
without using a calibrating pattern. 

Another object of the present invention is to 
provide an image constructor, an image constructing 
method, and a recording medium having an image 
constructing program recorded thereon, in which a 
UJ high precision panoramic image is obtained. 

w 

.3 In a lens distortion factor calculating 

apparatus for subjecting an image picked up by image 

s w 

pick-up means having a lens to lens distortion 
correction, a lens distortion factor calculating 
apparatus according to the present invention is 
characterized by comprising first means for finding, 
on the basis of two images picked up by the image 
pick-up means, the coordinates of a plurality of 
corresponding points between the images; second 
means for calculating, on the basis of the 
coordinates of the corresponding points found by the 
first means, geometric transform factors between the 
two images; and third means for calculating, on the 
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basis of the coordinates of the corresponding points 
found by the first means and the geometric transform 
factors found by the second means, a lens distortion 
factor . 

An example of the first means is one comprising 
means for extracting an overlapped portion of the 
two images picked up by the image pick-up means, 
means for extracting, from the overlapped portion 
of one of the images with the other image, a plurality 
of partial images effective for tracking by an 
optical flow between both the images as feature 
points, and means for tracking a point, which 
corresponds to each of the feature points on the one 
image, on the other image on the basis of the optical 
flow between both the images. 

In a lens distortion factor calculating method 
for subjecting an image picked up by image pick- 
up means having a lens to lens distortion correction, 
a lens distortion factor calculating method 
according to the present invention is characterized 
by comprising a first step of finding, on the basis 
of two images picked up by the image pick-up means, 
the coordinates of a plurality of corresponding 
points between the images; a second step of 
calculating, on the basis of the coordinates of the 

1 2 



corresponding points found in the first step, 



geometric transform factors between the two images; 



and a third step of calculating, on the basis of the 



coordinates'of the corresponding points found in the 



first step and the geometric transform factors found 



in the second step, a lens distortion factor. 



An example of the first step is one comprising 



the steps of extracting an overlapped portion of the 



m two images picked up by the image pick-up means, 

Si 

% extracting, from the overlapped portion of one of 

Bj the images with the other image, a plurality of 



ul partial images effective for tracking by an optical 

<7\ flow between both the images as feature points, and 

tracking a point, which corresponds to each of the 

\ 

Gl feature points on the one image, of the other image 

& on the basis of the optical flow between both the 



images . 



A computer readable recording medium having a 



lens distortion factor calculation program recorded 



thereon according to the present invention is a 



computer readable recording medium having a lens 



distortion factor calculation program for 



subjecting an image picked up by image pick-up means 



having a lens to lens distortion correction recorded 



thereon, characterized in that the lens distortion 
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factor calculation program causes a computer to 
carry out a first step of finding, on the basis of 
two images picked up by the image pick-up means, the 
coordinates of a plurality of corresponding points 
between the images; a second step of calculating, 
on the basis of the coordinates of the corresponding 
points found in the first step, geometric transform 
factors between the two images; and a third step of 
calculating, on the basis of the coordinates of the 
corresponding points found in the first step and the 
geometric transform factors found in the second 
step, a lens distortion factor. 

An example of the first step is one comprising 
the steps of extracting an overlapped portion of the 
two images picked up by the image pick-up means, 
extracting, from the overlapped portion of one of 
the images with the other image, a plurality of 
partial images effective for tracking in an optical 
flow between both the images as feature points, and 
tracking a point, which corresponds to each of the 
feature points on the one image, on the other image 
on the basis of the optical flow between both the 
images . 

In an image constructor for combining a first 
image and a second image which are picked up by image 
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pick-up means having a lens, an image constructor 
according to the present invention is characterized 
by comprising first means for finding, on the basis 
of the first image and the second image, the 
coordinates of a plurality of corresponding points 
between the images; second means for calculating, 
on the basis of the coordinates of the corresponding 
points found by the first means, geometric transform 
factors between the first image and the second image; 
third means for calculating, on the basis of the 
coordinates of the corresponding points found by the 
first means and the geometric transform factors 
found by the second means, a lens distortion factor; 
fourth means for subjecting the first image and the 
second image to lens distortion correction on the 
basis of the lens distortion factor calculated by 
the third means; and fifth means for combining the 
first image and the second image, which have been 
subjected to the lens distortion correction, 
obtained by the fourth means using the geometric 
transform factors between the first image and the 
second image which have been subjected to the lens 
distortion correction. 

An example of the first means is one comprising 
means for extracting an overlapped portion of the 
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first image and the second image, means for 
extracting, from the overlapped portion of one of 
the images with the other image, a plurality of 
partial images effective for tracking by an optical 
flow between both the images as feature points, and 
means for tracking a point, which corresponds to each 
of the feature points on the one image, on the other 
image on the basis of the optical flow between both 
the images. 

In an image construcging method for combining 
a first image and a second image which are picked 
up by image pick-up means having a lens, an image 
constructing method according to the present 
invention is characterized by comprising a first 
step of finding, on the basis of the first image and 
the second image, the coordinates of a plurality of 
corresponding points between the images; a second 
step of calculating, on the basis of the coordinates 
of the corresponding points found in the first step, 
geometric transform factors between the first image 
and the second image; a third step of calculating, 
on the basis of the coordinates of the corresponding 
points found in the first step and the geometric 
transform factors found in the second step, a lens 
distortion factor; a fourth step of subjecting the 



first image and the second image to lens distortion 
correction on the basis of the lens distortion factor 
calculated in the third step; and a fifth step of 
combining the first image and the second image, which 
have been subjected to the lens distortion 
correction, obtained in the fourth step using the 
geometric transform factors between the first image 
and the second image which have been subjected to 
the lens distortion correction. 

An example of the first step is one comprising 
the steps of extracting an overlapped portion of the 
first image and the second image, extracting, from 
the overlapped portion of one of the images with the 
other image, a plurality of partial images effective 
for tracking by an optical flow between both the 
images as feature points, and tracking a point, which 
corresponds to each of the feature points on the one 
image, on the other image on the basis of the optical 
flow between both the images. 

A computer readable recording medium having an 
image constructing program according to the present 
invention is a computer readable recording medium 
having an image constructing program for combining 
a first image and a second image which are picked 
up by image pick-up means having a lens recorded 



thereon, characterized in that the image 
constructing program causes a computer to carry out 
a first step of finding, on the basis of the first 
image and the second image, the coordinates of a 
plurality of corresponding points between the 
images; a second step of calculating, on the basis 
of the coordinates of the corresponding points found 
in the first step, geometric transform factors 
between the first image and the second image; a third 
step of calculating, on the basis of the coordinates 
of the corresponding points found in the first step 
and the geometric transform factors found in the 
second step, a lens distortion factor; a fourth step 
of subjecting the first image and the second image 
to lens distortion correction on the basis of the 
lens distortion factor calculated in the third step; 
and a fifth step of combining the first image and 
the second image, which have been subjected to the 
lens distortion correction, obtained in the fourth 
step using the geometric transform factors between 
the first image and the second image which have been 
subjected to the lens distortion correction. 

An example of the first step is one comprising 
the steps of extracting an overlapped portion of the 
first image and the second image, extracting, from 
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the overlapped portion of one of the images with the 
other image, a plurality of partial images effective 
for tracking by an optical flow between both the 
images as feature points, and tracking a point, which 
corresponds to each of the feature points on the one 
image, on the other image on the basis of the optical 
flow between both the images. 

The foregoing and other objects, features, 
aspects and advantages of the present invention will 
become more apparent from the following detailed 
description of the present invention when taken in 
conjunction with the accompanying drawings. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a schematic view for explaining a 

hierarchical estimation method, which illustrates 
an image in a hierarchy 1; 

Fig. 2 is a schematic view for explaining a 

hierarchical estimation method, which illustrates 
an image in a hierarchy 2; 

Fig. 3 is a schematic view for explaining a 

hierarchical estimation method, which illustrates 
an image in a hierarchy 3; 

Fig. 4 is a schematic view for explaining a 

hierarchical estimation method, which illustrates 
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an image in a hierarchy 4 ; 

Fig. 5 is a schematic view for explaining 
dilation processing performed in an optical flow 
estimation method employed in the present 
embodiment ; 

Fig. 6a is a schematic view showing an example 
of an optical flow which has been subjected to 
threshold processing for an image in a hierarchy, 
and Fig. 6b is a schematic view showing an optical 
flow after completion; 

Fig. 7 is a block diagram showing the 
configuration of a panoramic image constructor; and 

Fig. 8 is a flow chart showing the procedure for 
panoramic image constructing processing; 

Fig. 9 is a flow chart showing the detailed 
procedure for processing in the step 2 shown in Fig. 
8 ; and 

Fig. 10 is an illustration for explaining a 
planar projective transform. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Description is now made of an embodiment in a 
case where the present invention is applied to a 
panoramic image constructor. 

[1] Description of Panoramic Image Constructor 
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Fig. 7 illustrates the configuration of a 
panoramic image constructor. 

The panoramic image constructor is realized by 
a personal computer. A display 21, a mouse 22, and 
a keyboard 23 are connected to a personal computer 
10. The personal computer 10 comprises a CPU 11, 
a memory 12, a hard disk 13, a drive (a disk drive) 
14 of a removable disk such as CD-ROM. 

The hard disk 13 stores a panoramic image 
constructing processing program in addition to OS 
(Operating System) . The panoramic image 
constructing processing program is installed in the 
hard disk 13 using a CD-ROM 20 storing the program. 
It is assumed that a plurality of images picked up 
by a digital camera are previously stored in the hard 
disk 13. 

[2] Description of Panoramic Image 
Constructing Processing Performed by CPU 11 in Case 
Where Panoramic Image Constructing Program is 
Started 

Fig. 8 shows the procedure for overall 
processing performed by the CPU 11. 

For convenience of illustration, description 
is herein made of a case where two images are combined 
to construct a panoramic image. 



Two images to be combined are first read in the 
memory 12 (step 1) . 

Geometric transform factors between the read 
two images are calculated (step 2) . 

A lens distortion factor is then calculated 
(step 3) . In this case, geometric transform 
parameters are corrected on the basis of the lens 
distortion factor. 

The two images are subjected to lens distortion 
correction processing on the basis of the calculated 
lens distortion factor (step 4) . 

The two images which have been subjected to the 
lens distortion correction processing are combined 
on the basis of the two images and the geometric 
transform parameters corrected on the basis of the 
lens distortion factor (step 5) . 

Each of the steps 2 to 5 will be described in 
more detail. 

[3] Description of Processing in Step 2 Shown 
in Fig. 8 

Fig. 9 shows the detailed procedure for the 
processing in the step 2 shown in Fig. 8. 

For convenience of illustration, description 
is herein made of a case where two images which have 
an overlapped portion (a first image Al and a second 



image A2) are joined to each other. 

First, extraction processing of the overlapped 
portion of the first image Al and the second image 
A2 is performed (step 11) . The extraction processing 
of the overlapped portion is performed on the basis 
of the SSD (Sum of Squared Difference) method or the 
normalized correlation method, for example. 



In the SSD method, with respect to two original 
images Al and A2 whose overlapped portion should be 
extracted, images Ii and I 2 having lower resolutions 
than those of the original images are first 
respectively produced. An overlapped portion <w 
(size : M X N) of the two images Ii and I 2 having 
low resolutions is found using a squared error E per 
pixel, as expressed by the following equation (10) . 
An amount of movement (d) between the images is 
changed in an allowable range, and the overlapped 
portion is extracted from the amount of movement (d) 
in a case where E is the smallest: 



(a) Description of SSD method 



E(d) = 



(10) 



MxN 



(b) Description of Normalized 



Cross- 



correlation Method 



In the normalized c r o s s - c o r r e 1 a t i o n method. 



with respect to two original images Al and A2 whose 
overlapped portion should be extracted, images Ii 
and 1 2 having lower resolutions than those of the 
original images are first respectively produced. 
The overlapped portion co (size : M X N) of the two 
images Ii and I 2 having low resolution is found using 
a normalized c r o s s - c o r r e 1 a t i o n factor C, as 
expressed by the following equation (11) . An amount 
of movement (d) between the images is changed in an 
allowable range, and the overlapped portion is 
extracted from the amount of movement (d) in a case 
where C is the largest: 

Y (//x)-7J//x+cU-L) 



% In the equation (11), I i and I 2 are averages 

of respective gray scale values of a first image I i 

and a second image I 2 in the overlapped portion of 
both the images in a case where the first image is 
fixed and the second images is moved by d. <x i and 
a 2 are respective variances of the gray scale values 
of the first image Ii and the second image I 2 in the 
overlapped portion of both the images in a case where 
the first image Ii is fixed and the second image I 2 
is moved by d. 

Feature point extraction processing is then 



2 4 



performed (step 12) . That is, a plurality of partial 
images (rectangular regions) which are effective for 
tracking are extracted as feature points from the 
overlapped portion of the first image Al with the 
second image A2. The feature points are extracted 
such that they are not overlapped with each other. 
Specifically, the above-mentioned portion having a 
high eigenvalue X min (see the equation (8)) is 
extracted as the feature point. 

Feature point tracking processing is then 
performed (step 13). That is, a position, on the 
second image A2, of the extracted feature point on 
the first image Al is tracked. 

Specifically, an optical flow vector for each 
patch of suitable size (13 X 13, for example) is 
first found by the optical flow estimating method 
developed by the applicant of the present invention 
(see [2] in column * Description of the Prior Art") . 
The position, on the second image A2 corresponding 
to the feature point on the first image Al is found 
in units not exceeding units of pixels by linear 
interpolation from flow vectors in the patches in 
four vicinities of the feature point on the first 
image Al. Consequently, in the overlapped portion 
of the first image Al and the second image A2, the 
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coordinates of each of corresponding points on both 
the images are obtained. 

A geometric transform matrix (geometric 
transform factors) is then calculated (step 14) . 

As the geometric transform factors, a planar 
projective transform matrix is used herein. Affine 
transformation, 2D rigid transform, 2D translation, 
and so forth may be used as geometric transformation 
depending on imaging conditions. 

When a scene to be an object is far, or is a wall 
of a building or a blackboard, or the like even if 
it is near, it can be assumed that the scene exists 
on a simple plane. It is known in projective geometry 
that when a point M on a single plane in a 
three-dimensional space is observed from two 
different view points Ci and C 2 , as shown in Fig. 10, 
transformation between coordinates mi and m 2 on their 
respective image surfaces is linear, which is 
referred to as homography (O. Faugeras. "Three- 
Dimension Computer Viaion: a Geometric Viewpoint 
HIT press, 1993.). 

That is, in a case where the coordinates of an 
image are represented by homogeneous coordinates, 
a point m 2 = (x 2 , Y2 r 1 ) * ° n the second image has a 
corresponding point mi = (x if yi , l) fc on the first 
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image, and the relationship therebetween is defined 
by the following equation (12) : 



m l ~ Hm 2 = 



( K K K 

K K h 4 



'4 



1 



A 1 J 



(12) 



~ indicates projective equivalence, leaving a 
scale factor. The transform matrix H can be 
rewritten, as expressed by the following equation 
( 13 ) : 

h 0 x 2 +h l y 2 +h 2 _ 

(13) 



x, = 



h 6 x 2 +h 7 y 2 +l 
h 3 x 2 +h A y 2 +h 5 



Jl h 6 x 2 +h 1 y 2 +l 
[4] Description of Processing (Lens Distortion 
Factor Calculation Processing) in Step 3 in Fig. 8 
Lens distortion means such a phenomenon that a 
lens is geometrically distorted depending on the 
distance from the center of lens distortion. 
Coordinates on an image plane shall be referred to 
as image coordinates. Letting (u t , v t ) be true image 
coordinates including no lens distortion, actually 
observed image coordinates (u a , v a ) including lens 
distortion can be generally expressed by the 
following equation (14) : 



u a =u t +(u t ~u 0 )kr 2 

v a = v f +(v,-v 0 )£r 2 - (14) 

r 2 =(«,-w 0 ) 2 +(w f ™v 0 ) 2 

In the foregoing equation (14) k is a 
distortion factor, and (u Q , v Q ) are the coordinates 
at the center of lens distortion. 

Letting Pi be the coordinates of each feature 
point in the first image Al, P'i be coordinates 
corresponding thereto on the second image A2, and 
H be a geometric transform matrix between both the 
images, an error function E can be defined, as 
expressed by the following equation (15) : 



* = £(p f -flW) 2 - (is) 



If it is assumed that the coordinate (u Q , v Q ) 
at the center of lens distortion coincides with the 
center of an image, the error function E can be 
rewritten, as expressed by the following equation 
(16) , using the distortion factor k in the foregoing 
equation (14) : 



The lens distortion factor k for minimizing the 
error E is calculated using the N e w t o n - R a p h s o n 
method. That is, in a case where a variance dk of 
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the factor k is very small, when the terms of second 
and higher degrees are ignored by Tailor series 
expansion, the following equation (17) is obtained: 
E(k + dk) = E{k) + dkE\k) (17) 

where E 9 is a linear differential of the error 
function E. Since it is desired to find the minimum 
value of E, the following equation (18) is obtained: 

E(k + dk) = Q — (18) 

When the foregoing equation (18) is substituted 
in the foregoing equation (17) , dk can be calculated 
by the following equation (19) : 

dk = — (19) 

E\k) 

Furthermore, dk can be calculated with higher 
precision by iteration, as expressed by the 
foregoing equation (20) : 

dk & IQA ... (20) 

That is, dk 0 , dki, dk 2/ — dk n are successively 
found, taking a starting point as k = 0. When dk 0 
is calculated, a matrix (which is taken as H 0 ) 
calculated in the step 2 is used as the geometric 
transform matrix H in the foregoing equation (16) . 
When dk 0 is found, corresponding points (feature 
points) between the first image Al and the second 
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image A2 which are used for calculating the geometric 
transform matrix H are subjected to lens distortion 
correction on the basis of the foregoing equation 
(14), to newly calculate a geometric transform 
matrix Hi from the corresponding points which have 
been subjected to the lens distortion correction. 

In calculating d k i , the geometric transform 
matrix Hi newly calculated is used as the geometric 
transform matrix H in the foregoing equation (16) . 
The same is true for a case where dk 2 and later are 
calculated. A geometric transform matrix H n is newly 
calculated on the basis of dk n (= k) finally found. 

In finding dk 0 , dki, dk 2 , d k „ , the matrix H 

calculated in the step 2 may be always used as the 
geometric transform matrix H in the foregoing 
equation (16). Also in this case, the geometric 
transform matrix H n is newly calculated on the basis 
of dk n (= k) finally found. 

[5] Description of Processing (Lens Distortion 
Correction Processing) in Step 4 in Fig. 8 

In the lens distortion correction processing in 
the step 4, images Al' and A 2 in which lens 
distortion has been corrected are produced from the 
first image Al and the second image A2 on the basis 
of the lens distortion factor dk (= k) finally found 
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in the step 3 and the foregoing equation (14) . 

That is, letting <u t , v t ) be a predetermined 
position of true image coordinates, image 
coordinates including lens distortion 

corresponding to the predetermined position (u t , v t ) 
are found from the foregoing equation (14). Image 
data representing the image coordinates including 
lens distortion corresponding to the predetermined 
position <u t , v t ) shall be image data representing 
the predetermined position (u t , v t ) • 

[6] Description of Processing (Image Combining 
Processing) in Step 5 in Fig. 8 

In the image combining processing in the step 
5, the first image Al' and the second image A2 ' , in 
which lens distortion has been corrected, obtained 
in the step 4 are combined on the basis of the first 
image Al' and the second image A2' and the geometric 
transform matrix H n calculated on the basis of dk n 
(= k ) finally found in the step 3. 

Although the present invention has been 
described and illustrated in detail, it is clearly 
understood that the same is by way of illustration 
and example only and is not to be taken by way of 
limitation, the spirit and scope of the present 
invention being limited only by the terms of the 



appended claims 
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